Method and electronic device for convolution calculation in neutral network

ABSTRACT

Disclosed is a method for convolution calculation in a neural network, comprising: reading an input feature map, depthwise convolution kernels and pointwise convolution kernels from a dynamitic random access memory (DRAM); performing depthwise convolution calculations and pointwise convolution calculations according to the input feature map, the depthwise convolution kernels and the pointwise convolution kernels to obtain output feature values of a first predetermined number p of points on all pointwise convolution output channels; storing the output feature values of a first predetermined number p of points on all pointwise convolution output channels into an on-chip memory, wherein the first predetermined number p is determined according to at least one of available space in the on-chip memory, a number of the depthwise convolution calculation units, and width, height and channel dimensions of the input feature map; and repeating the above operation obtain output feature values of all points on all pointwise convolution output channels. Therefore, the storage space for storing intermediate results may be reduced.

TECHNICAL FIELD OF THE DISCLOSURE

The present disclosure relates to the technical field of neural network,and more particularly to a method and an electronic device forconvolution calculation in a neural network.

BACKGROUND

Deep learning technology based on convolutional neural network may beused for image recognition and detection, speech recognition and so onwith high accuracy, so it is widely used in the fields of safetymonitoring, auxiliary driving, intelligent companion robot, intelligentmedical treatment and so on.

Since the convolutional neural network is operation intensive, reducingparameters amount and calculation amount of neural network has become ahot direction in current research. A Mobile network (i.e. MobileNet) isa latest special convolutional neural network, which reduces thecalculation amount by decomposing the traditional three-dimensionalconvolution operation into two convolution operations, i.e. depthwiseconvolution and pointwise convolution, while the calculation accuracy islittle different from that of the traditional convolution.

SUMMARY

Regarding the existing implementation solution of MobileNet, whether itis based on a general purpose processor (CPU), a dedicated graphicsprocessor (GPU), or a dedicated processing chip, it is necessary tofirstly calculate the output of depthwise convolution operation, andthen take them as input data of the pointwise convolution operation, andthen perform calculations.

The disadvantage of this approach is that when the data amount of inputand output is relatively large, a larger on-chip random access memory(SRAM) is required for buffering intermediate results. However, a sizeof on-chip SRAM is fixed. If the size of on-chip SRAM is insufficient tobuffer the intermediate results, it is necessary to split the depthwiseconvolution operation into multiple calculations and write eachcalculation result into off-chip memory (DDR) until the calculationresults of the depthwise convolution operation are completely calculatedand written into the off-chip memory (DDR), and then read these resultsout of DDR in batches and perform pointwise convolution calculations.Obviously, this will place a huge burden on the limited datatransmission bandwidth and lead to an increase in system powerconsumption.

In order to solve the above technical problem, the present disclosure isproposed. A method and an electronic device for convolution calculationin a neural network which may reduce a storage space for storingintermediate results are provided in embodiments of the presentdisclosure.

According to an aspect of the present disclosure, disclosed is a methodfor convolution calculation in a neural network comprising: reading aninput feature map, depthwise convolution kernels and pointwiseconvolution kernels from a dynamitic random access memory (DRAM);performing the depthwise convolution calculations and pointwiseconvolution calculations according to the input feature map, thedepthwise convolution kernels and the pointwise convolution kernels toobtain output feature values of a first predetermined number p of pointson all pointwise convolution output channels, storing the output featurevalues of a first predetermined number p of points on all pointwiseconvolution output channels into an on-chip memory, wherein the firstpredetermined number p is determined according to at least one ofavailable space in the on-chip memory, a number of the depthwiseconvolution calculation units, and width, height and channel dimensionsof the input feature map, and repeating above operation to obtain outputfeature values of all points on all pointwise convolution outputchannels.

In one embodiment, performing depthwise convolution calculations andpointwise convolution calculations according to the input feature map,the depthwise convolution kernels and the pointwise convolution kernelsto obtain output feature values of a first predetermined number p ofpoints on all pointwise convolution output channels comprises:performing the depthwise convolution calculations according to the inputfeature map and the depthwise convolution kernels to obtain intermediatefeature values of the first predetermined number p of points on alldepthwise convolution output channels; and performing the pointwiseconvolution calculations according to the intermediate feature values ofthe first predetermined number p of points on all depthwise convolutionoutput channels and the pointwise convolution kernels, to obtain outputfeature values of the first predetermined number p of points on allpointwise convolution output channels.

In one embodiment, performing depthwise convolution calculations andpointwise convolution calculations according to the input feature map,the depthwise convolution kernels and the pointwise convolution kernelsto obtain output feature values of the first predetermined number p ofpoints on all pointwise convolution output channels comprises:performing the depthwise convolution calculations according to the inputfeature map and the depthwise convolution kernels, to obtainintermediate feature values of the first predetermined number p ofpoints on a second predetermined number m of depthwise convolutionoutput channels; performing the pointwise convolution calculationsaccording to the intermediate feature values of the first predeterminednumber p of points on the second predetermined number m of depthwiseconvolution output channels and the pointwise convolution kernels, toobtain a current pointwise convolution partial sums of the firstpredetermined number p of points on all the pointwise convolution outputchannels; respectively performing accumulation calculations on thecurrent pointwise convolution partial sums of the first predeterminednumber p of points on all pointwise convolution output channels andprevious accumulation calculation results of the first predeterminednumber p of points, to generate current accumulation calculation resultsof the first predetermined number p of points; and repeating aboveoperations, performing the pointwise convolution calculations accordingto intermediate feature values of the first predetermined number p ofpoints on a next second predetermined number m of depthwise convolutionoutput channels and the pointwise convolution channel, andcorrespondingly performing subsequent operations, until the pointwiseconvolution calculations and accumulation calculations are completed onall of intermediate feature values of the first predetermined number pof points on all depthwise convolution output channels, the finalaccumulation calculation results of the first predetermined number p ofpoints being the output feature values of the first predetermined numberp of points on all pointwise convolution output channels.

According to another aspect of the present disclosure, disclosed is anelectronic device comprising a processor, and a memory having computerprogram instructions stored therein, when executed by the processor,making the processor to perform a method for convolution calculation ina neutral network comprising: reading an input feature map, depthwiseconvolution kernels and pointwise convolution kernels from a dynamiticrandom access memory (DRAM); performing depthwise convolutioncalculations and pointwise convolution calculations according to theinput feature map, the depthwise convolution kernels and the pointwiseconvolution kernels to obtain output feature values of a firstpredetermined number p of points on all pointwise convolution outputchannels; storing the output feature values of a first predeterminednumber p of points on all pointwise convolution output channels into anon-chip memory, wherein the first predetermined number p is determinedaccording to at least one of available space in the on-chip memory, anumber of the depthwise convolution calculation units, and width, heightand channel dimensions of the input feature map; and repeating the aboveoperation to obtain output feature values of all points on all pointwiseconvolution output channels.

According to another aspect of the present disclosure, disclosed is acomputer program product comprising computer program instructions, whenexecuted by a processor, making the processor to perform a method forconvolution calculation in a neutral network comprising: reading aninput feature map, depthwise convolution kernels and pointwiseconvolution kernels from a dynamitic random access memory (DRAM);performing depthwise convolution calculations and pointwise convolutioncalculations according to the input feature map, the depthwiseconvolution kernels and the pointwise convolution kernels to obtainoutput feature values of a first predetermined number p of points on allpointwise convolution output channels; storing the output feature valuesof a first predetermined number p of points on all pointwise convolutionoutput channels into an on-chip memory, wherein the first predeterminednumber p is determined according to at least one of available space inthe on-chip memory, a number of the depthwise convolution calculationunits, and width, height and channel dimensions of the input featuremap; and repeating the above operation to obtain output feature valuesof all points on all pointwise convolution output channels.

According to another aspect of the present disclosure, disclosed is acomputer readable and writable storage medium having computer programinstructions stored thereon, when executed by a processor, making theprocessor to perform a method for convolution calculation in the neuralnetwork comprising: reading an input feature map, depthwise convolutionkernels and pointwise convolution kernels from a dynamitic random accessmemory (DRAM); performing depthwise convolution calculations andpointwise convolution calculations according to the input feature map,the depthwise convolution kernels and the pointwise convolution kernelsto obtain output feature values of a first predetermined number p ofpoints on all pointwise convolution output channels; storing the outputfeature values of a first predetermined number p of points on allpointwise convolution output channels into an on-chip memory, whereinthe first predetermined number p is determined according to at least oneof available space in the on-chip memory, a number of the depthwiseconvolution calculation units, and width, height and channel dimensionsof the input feature map; and repeating the above operation to obtainoutput feature values of all points on all pointwise convolution outputchannels.

Compared with the prior art, the convolution calculation method in theneural network and the electronic device according to embodiments of thepresent disclosure may perform depthwise convolution calculations andpointwise convolution calculations according to an input feature map,depthwise convolution kernels and pointwise convolution kernels toobtain output feature values of the first predetermined number p ofpoints on all pointwise convolution output channels, and repeat theabove operations to obtain output feature values of all points on allpointwise convolution output channels. Therefore, the storage space forstoring intermediate results may be reduced, such that a more efficientconvolutional neural network may be realized.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentdisclosure will become more obvious through describing the embodimentsof the present disclosure in more detail with reference to accompanyingdrawings. The drawings are used to provide a further understanding tothe embodiments of the present disclosure and constitute a portion ofthe specification, and the drawings, together with the embodiments ofthe present disclosure, are used to explain this disclosure and do notconstitute restrictions on the disclosure. In the drawings, the samereference number generally refers to the same portion or step.

FIG. 1 shows a schematic diagram of a convolution kernel in aconventional convolutional neural network.

FIG. 2A shows a schematic diagram of a convolution kernel of a depthwiseconvolution operation in a MobileNet.

FIG. 2B shows a schematic diagram of a convolution kernel of a pointwiseconvolution operation in a MobileNet.

FIG. 3 shows a flowchart of a method for convolution calculationaccording to an embodiment of the present disclosure.

FIG. 4 shows a flowchart of the step of calculating output featurevalues of a first predetermined number p of points on all pointwiseconvolution output channels according to the first embodiment of thepresent disclosure.

FIG. 5 shows a flowchart of the step of calculating intermediate featurevalues of a first predetermined number p of points on all depthwiseconvolution output channels according to the first embodiment of thepresent disclosure.

FIG. 6 shows a schematic diagram of the step of calculating intermediatefeature values of a first predetermined number p of points on alldepthwise convolution output channels according to the first embodimentof the present disclosure.

FIG. 7 shows a flowchart of the step of calculating output feature valueof a first predetermined number p of points on all pointwise convolutionoutput channels according to the intermediate feature values and thepointwise convolution kernels according to the first embodiment of thepresent disclosure.

FIG. 8 shows a schematic diagram of the step of calculating outputfeature values of a first predetermined number p of points on allpointwise convolution output channels according to the intermediatefeature values and the pointwise convolution kernels according to thefirst embodiment of the present disclosure.

FIG. 9 shows a schematic diagram of depthwise convolution calculationsaccording to an embodiment of the present disclosure.

FIG. 10 shows a schematic diagram of pointwise convolution calculationsaccording to an embodiment of the present disclosure.

FIG. 11 shows a flowchart of the step of calculating the output featurevalue of a first predetermined number p of points on all pointwiseconvolution output channels according to the intermediate feature valuesand the pointwise convolution kernels according to the second embodimentof the present disclosure.

FIG. 12 shows a flowchart of the step of calculating output featurevalues of a first predetermined number p of points on all pointwiseconvolution output channels according to the intermediate feature valuesand the pointwise convolution kernels according to the second embodimentof the present disclosure.

FIG. 13 shows a flowchart of the step of calculating output featurevalues of a first predetermined number p of points on all pointwiseconvolution output channels according to the third embodiment of thepresent disclosure.

FIG. 14 shows a schematic diagram of the step of calculating a currentpointwise convolution partial sums of a first predetermined number p ofpoints on all pointwise convolution output channels according to thethird embodiment of the present disclosure.

FIG. 15 shows a block diagram of an electronic device according to anembodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, an exemplary embodiment according to the present disclosurewill be described in detail with reference to the drawings. Obviously,the described embodiments are only a portion of embodiments of thepresent disclosure and not all the embodiments of the presentdisclosure, and it should be understood that the present disclosure isnot limited by the exemplary embodiment described herein.

SUMMARY OF THE DISCLOSURE

A convolutional neural network may generally include multipleconvolutional layers. In each convolutional layer, a convolution kernelof the layer is used to perform convolution operations on the inputfeature map (also known as input feature data or input feature value) ofthe layer to obtain an output feature map (also known as output featuredata or output feature value) of the layer. In each layer of theconvolutional neural network, the input feature map may have a certainwidth and height, and may have a certain number of channels (also knownas depth). Each convolution kernel may have the same (or different)width and height, which are less than (or equal to) the width and heightof the input feature map, and may have the same number of channels,which is equal to the number of channels of the input feature map.

As a lightweight neural network, a MobileNet uses the idea of depthwiseseparable convolutions, and instead of fusing channels when calculatingconvolutions (e.g., 3*3 convolution kernel or larger size), it usesdepthwise (or known as channel-wise) and 1*1 pointwise convolutionmethod to decompose convolution, such that the speed and model size areoptimized, and the calculation accuracy is basically kept.

Next, a comparison between the convolution calculation process of theconventional convolutional neural network and that of the MobileNet willbe described with reference to FIG. 1 to FIG. 2B.

FIG. 1 shows a schematic diagram of convolution kernels in aconventional convolutional neural network, FIG. 2A shows a schematicdiagram of convolution kernels of depthwise convolution operations inthe MobileNet, and FIG. 2B shows a schematic diagram of convolutionkernels of pointwise convolution operations in the MobileNet.

As shown in FIG. 1, assuming that the size of convolution kernels in aconventional convolutional neural network is R rows and S columns, witha total of M channels and N such convolution kernels, then thecalculation amount of convolution calculation with the input feature mapto output one point is R*S*M*N. Assuming that the input feature map sizeis X rows and Y columns, with a total of M channels, then thecalculation amount of the entire convolution calculation is R*S*M*N*X*Y.Expressed by a formula, the value of a certain output point (x,y,n) is:

${Fout}_{x,y,n} = {\sum\limits_{r,s,m}{K_{r,s,m,n} \star {Fin}_{{x + s - 1},{y + r - 1},m}}}$

The cascade of the depthwise convolution shown in FIG. 2A and thepointwise convolution shown in FIG. 2B is a MobileNet convolutionoperation corresponding to the conventional convolution operation inFIG. 1.

The depthwise convolution in FIG. 2A may be regarded as splitting Mchannels of one convolution kernel in the conventional convolution intoM depthwise convolution kernels, each of which has R rows and S columns,and only 1 channel. The M depthwise convolution kernels convolve withthe M channels of the input feature map, respectively, to obtain outputresults of the M channels without accumulation. The calculation amountof the entire convolution calculation is R*S*M*1*X*Y. Expressed by aformula, the value of a certain output point (x,y,m) is:

${Fout}_{x,y,n} = {\sum\limits_{r,s}{K_{r,s,m}^{\prime} \star {Fin}_{{x + s - 1},{y + r - 1},m}}}$

The pointwise convolution in FIG. 2B is exactly the same as theconventional convolution operation, except that the size of theconvolution kernel is 1 row and 1 column, with a total of M channels andN of such convolution kernels. The N depthwise convolution kernelsrespectively convolve with the input feature map to obtain outputresults of the N channels. The calculation amount of the entireconvolution calculations is 1*1*M*N*X*Y. Expressed by a formula, thevalue of a certain output point (x,y,n) is:

${Fout}_{x,y,n} = {\sum\limits_{m}{K_{m,n}^{\prime\prime} \star {Fin}_{{x - 1},{y - 1},m}}}$

It may be seen that the MobileNet convolution operation reduces thecalculation amount of conventional convolution from R*S*M*N*X*Y toR*S*M*X*Y+M*N*X*Y, which significantly reduces the calculation amount ofconvolution operations. Therefore, it may be seen that in the case whereR*S is 3*3, the calculation amount is one-ninth to one-eighth of theequivalent conventional convolution.

At present, all of the existing implementation solutions of MobileNetneed to firstly calculate intermediate output results of the depthwiseconvolution operations, and store them continuously in an on-chip SRAMuntil all calculations are completed, and then read them from theon-chip SRAM as input data of the pointwise convolution operation, andperform calculation. Since a large number of depthwise convolutionintermediate output results are to be stored in the on-chip randomaccess memory, a large amount of on-chip memories are required, whichcauses an increase in chip area and cost, or if the intermediate resultsare stored in an off-chip random access memory, it will bring a greaterburden on the limited data transmission bandwidth and increase the powerconsumption of the system.

For this technical problem, considering the special convolutionstructure of MobileNet network-depthwise convolution followed bypointwise convolution, the present disclosure provides a method forconvolution calculation in a neural network, an apparatus, an electronicdevice, a computer program product and a computer readable storagemedium, which may perform a depthwise convolution calculation and apointwise convolution calculation according to an input feature map,depthwise kernels and pointwise kernels to obtain output feature valuesof a first predetermined number p of points on all of pointwise outputchannels, and repeat the above operation to obtain output feature valuesof all the points on all the pointwise output channels. Therefore, thestorage space for storing the intermediate output results of thedepthwise convolution may be reduced, such that a more efficientconvolutional neural network may be realized.

Those skilled in the art should understand that the convolutioncalculation method according to embodiments of the present disclosuremay be applied not only to the MobileNet convolutional neural network,but also to other types of convolutional neural networks as long as theyinclude a convolution calculation process with a depthwise convolutionfollowed by a pointwise convolution, therefore, the embodiments of thepresent disclosure are not intended to impose any limitation on the typeof convolutional neural network.

After introducing the basic principles of the present disclosure,various non-limiting embodiments of the present disclosure will bespecifically described below by taking MobileNet as an example andreferring to the drawings.

Exemplary Method

FIG. 3 shows a flowchart of a method for convolution calculationaccording to an embodiment of the present disclosure.

As shown in FIG. 3, the convolution calculation method according to anembodiment of the present disclosure may comprise:

in step S110, a depthwise convolution calculation and a pointwiseconvolution calculation are performed according to an input feature map,depthwise convolution kernels and pointwise convolution kernels, toobtain output feature values of a first predetermined number p of pointson all pointwise convolution output channels.

The first predetermined number p may be determined according to at leastone of available space in a memory, a number of depthwise convolutioncalculation units, height and width dimension, and channel numberdimensions of the input feature map.

For example, the memory may be an on-chip random access memory (SRAM) toachieve faster access speed and avoid occupying data transmissionbandwidth. However, the present disclosure is not limited thereto. Forexample, the memory may be other memories, such as an off-chip memory(DDR). The available space in the memory may be used to bufferintermediate output results of depthwise convolution operations.

For example, when the width and height are larger (i.e. the points aremore) and the number of channels of the current convolutional layer issmaller (i.e. the depth is very shallow) (e.g., the current layer is inthe first few layers of the entire convolutional neural network), thefirst predetermined number p may be set to a larger value. On thecontrary, when the width and height of the current convolutional layerare smaller (i.e. the points are fewer) and the number of channels islarger (i.e. the depth is very deep) (e.g., the current layer is in thelatest few layers of the entire convolutional neural network), the firstpredetermined number p may be set to a smaller value.

Assuming that the depthwise convolution has M channels (or also referredto as M depthwise convolution kernels), the available space in thememory needs to buffer the intermediate output results of the depthwiseconvolution operations, i.e. the p*M intermediate feature values need tobe buffered. When the feature values are unquantized data, the capacityC of the available space should be greater than or equal to p*m*32 bits,and when the feature values are 8-bit quantized data, the capacity C ofthe available space should be greater than or equal to p*m*8 bits.Therefore, it may be seen from another angle that in a case where thecapacity C of the available space is fixed, if quantization is not used,the first predetermined number p may take C/(M*32); if quantization isused, the first predetermined number p may take C/(M*8).

Furthermore, the first predetermined number p may also be limited by thenumber of depthwise convolution calculation units (e.g.,multiplier-adder unit MAC), which may be a divisor of the number ofmultiplier-adder units.

In step S120, the above operation (i.e. step S110) is repeated to obtainoutput feature values of all points on all pointwise convolution outputchannels.

For example, step S110 is performed again, and output feature values ofa next first predetermined number p of points on all pointwiseconvolution output channels are calculated, and step S110 iscontinuously repeated until the output feature values of all points onall pointwise convolution output channels are obtained.

If the number of remaining points is not enough to generate the outputfeature values of p points during the process of the last calculation,it may be realized by a padding (e.g., 0 is padded) manner.Alternatively, it may also be realized by other manners such as reducingworking convolution calculation units.

According to the existing implementation solutions, all intermediateoutput results of the depthwise convolution operations are firstlycalculated and stored. It is assumed that as the intermediate outputresults of the depthwise convolution operations, there are a total of Hrows and W columns in the height and width dimensions. In an embodimentof the present disclosure, according to the feature that the depthwiseconvolution is followed by the pointwise convolution in Mobilenet,instead of waiting for the calculation for all of the H*W intermediateoutput points having been completed before calculating the pointwiseconvolution operations, the pointwise convolution operations areperformed immediately after only the depthwise convolution results of p(p is less than H*W, preferably much less than H*W, certainly p may alsobe equal to H*W) points are calculated, such that the storage spacerequired for the intermediate output results of the depthwiseconvolution operation is reduced to p/(H*W) of the storage spacerequired for the conventional convolutional neural network, whichsignificantly reduces the storage space for storing the intermediateoutput results of the depthwise convolution operations. Assuming thatthe intermediate data amount output by the depthwise convolutionoperations is H*W*M, then in the embodiment of the present disclosure,only the on-chip storage resource having a p*M (may be much less thanH*W*M) size is needed, in order to avoid a complex process that writingthe depthwise convolution results to the off-chip memory in batches dueto insufficient on-chip storage space until the intermediate resultsoutput by all depthwise convolution operations have been calculated andthen are read in batches for pointwise convolution calculations.According to the statistics, in a case where the on-chip storageresources are not enough to store next full-layer of convolution outputresults, but p/(H*W) of the next full-layer of convolution outputresults may be stored, the present disclosure reduces the datatransmission bandwidth of the MobileNet network by about 50%.

Hereinafter, it will describe in detail in various embodiments the stepS110 of performing the depthwise convolution calculations and thepointwise convolution calculations in the input feature map to obtainthe output feature values of the first predetermined number p of pointson all pointwise convolution output channels according to theembodiments of the present disclosure.

It should be noted that various embodiments may also be combinedtogether in whole or in part where possible, although the variousembodiments has been respectively described.

First Embodiment

In the first embodiment of the present disclosure, in order to obtainthe output feature values of the first predetermined number p of pointson all pointwise convolution output channels, the following operationsmay be performed: (1) performing depthwise convolution operations toobtain intermediate feature values of a first predetermined number p ofpoints on all depthwise convolution output channels; (2) performingpointwise convolution operations based on the intermediate featurevalues to obtain output feature values of p points on one or morepointwise convolution output channels; (3) repeating the above operation(2) to obtain output feature values of p points on all pointwiseconvolution output channels.

FIG. 4 shows a flowchart of the step of calculating the output featurevalues of the first predetermined number p of points on all pointwiseconvolution output channels according to the first embodiment of thepresent disclosure.

As shown in FIG. 4, according to the first embodiment of the presentdisclosure, the step S110 of calculating the output feature values ofthe first predetermined number p of points on all pointwise convolutionoutput channels may comprise:

In step S210, depthwise convolution calculations are performed accordingto an input feature map and a depthwise convolution kernel to obtainintermediate feature values of a first predetermined number p of pointson all depthwise convolution output channels.

In the current convolutional layer, an input feature map and a depthwiseconvolution kernel are obtained, and depthwise convolution calculationsare performed. It is assumed that the size of the input feature map is Xrows and Y columns, and a total of M channels. Correspondingly, thereare M depthwise convolution kernels, and the size of each depthwiseconvolution kernel is R rows and S columns, and only 1 channel. Whenperforming the depthwise convolution calculation, the first channel ofthe input feature map is convolved with the first depthwise convolutionkernel to obtain a first channel of the intermediate feature map, andthe second channel of the input feature map is convolved with the seconddepthwise convolution kernel to obtain the second channel of theintermediate feature map, and so on, to obtain the intermediate featuremap containing the intermediate feature values and having H rows and Wcolumns and a total of M channels. H=R and W=S when a stride of thedepthwise convolution calculation is 1 and the padding is 1.

In the first embodiment, unlike the prior art, instead of all theintermediate feature values of all H*W points on all M depthwiseconvolution output channels having been directly calculated, only theintermediate feature values of p points on all M channels arecalculated.

In step S220, the pointwise convolution calculations are performedaccording to the intermediate feature values of the first predeterminednumber p of points on all depthwise convolution output channels andpointwise convolution kernels, to obtain output feature values of thefirst predetermined number p of points on all pointwise convolutionoutput channels.

The pointwise convolution kernels are obtained, and the pointwiseconvolution calculations are performed on the intermediate featurevalues of p points on all M channels, to obtain the output featurevalues of p points on all pointwise convolution output channels.

As described above, the size of intermediate feature map is H rows and Wcolumns, with a total of M channels. Correspondingly, there are Npointwise convolution kernels, each of which has 1 row and 1 column,with only 1 channel. When performing the pointwise convolutioncalculations, all channels of the intermediate feature map areconvoluted with all channels of the first pointwise convolution kernelto obtain the first channel of the output feature map, and all channelsof the intermediate feature map are convoluted with all channels of thesecond pointwise convolution kernel to obtain the second channel of theoutput feature map, and so on, to obtain the output feature mapincluding output feature values and having E rows and F columns, with atotal of N channels. When a stride of the pointwise convolutioncalculation is 1, E=H and F=W.

Firstly, the step S210 of performing the depthwise convolutioncalculation on the input feature map to obtain the intermediate featurevalues of the first predetermined number p of points on all depthwiseconvolution output channels will be described with reference to FIG. 5and FIG. 6.

FIG. 5 shows a flowchart of the step of calculating the intermediatefeature values of the first predetermined number p of points on alldepthwise convolution output channels according to the first embodimentof the present disclosure. FIG. 6 shows a schematic diagram of the stepof calculating the intermediate feature values of the firstpredetermined number p of points on all depthwise convolution outputchannels according to the first embodiment of the present disclosure.

As shown in FIG. 5, the step S210 of calculating the intermediatefeature values of the first predetermined number p of points on all thedepthwise convolution output channels may comprise:

In step S211, depthwise convolution calculations are performed accordingto the input feature map and the depthwise convolution kernel, to obtainintermediate feature values of the first predetermined number p ofpoints on a second predetermined number m of depthwise convolutionoutput channels.

For example, the second predetermined number m may be determinedaccording to the number of depthwise convolution calculation units andthe first predetermined number p. For example, based on computationalefficiency considerations, it is desirable to make the depthwiseconvolution calculation units in the hardware circuit operate at fullcapacity. In this case, the second predetermined number m*the firstpredetermined number p=the number of MACs of the depthwise convolutioncalculation units.

Assuming that there are 512 multiplier-adder units for depthwiseconvolution calculations, then, for example, the depthwise convolutioncalculations of 32 points (p=32), 16 channels (m=16) may besimultaneously calculated at a time. As mentioned above, differentvalues of p and m may also be selected for other considerations. Forexample, when the width and height of the current convolutional layerare relatively large and the number of channels is relatively small, pmay be set to a larger value, for example, 64, 128, etc., andcorrespondingly, m may be set to a smaller value, for example, 8, 4,etc. Conversely, when the width and height of the current convolutionallayer are smaller and the number of channels is larger, p may be set toa smaller value, for example, 16, 8, etc., and correspondingly, m may beset to a larger value, for example, 32, 64, etc.

For example, as shown in FIG. 6, the intermediate feature map (1) may befirstly calculated based on the input feature map (1) and the depthwiseconvolution kernel (1). In FIG. 6, the input feature map (1) is a set ofinput feature points which have p points, m input channels, and thedepthwise convolution kernel (1) is a set of m depthwise convolutionkernels.

In an example, the step S211 may comprise:

Substep 1: reading input feature values of the first predeterminednumber p group of points on the second predetermined number m of inputchannels of the input feature map. In an example, the input featurevalues of a first predetermined number p of groups of points on a secondpredetermined number m of input channels are read from the input featuremap, concurrent with the step of performing pointwise convolutioncalculations according to the intermediate feature values of the firstpredetermined number p of points on the second predetermined number m ofdepthwise convolution output channels and the pointwise convolutionkernels. In another example, the above substep of reading the inputfeature values of a first predetermined number p of groups of points ona second predetermined number m of input channels are not concurrentwith the above step of performing pointwise convolution calculationsaccording to the intermediate feature values of the first predeterminednumber p of points on the second predetermined number m of depthwiseconvolution output channels and the pointwise convolution kernels.

For example, the input feature values of the first predetermined numberp group of points on the second predetermined number m of input channelsmay be read from the input feature map (as shown in the input featuremap in FIG. 6), and each group of points has a width and a height (asshown in the shadow in the input feature map (1) in FIG. 6) equal to awidth and a height (as shown in the shadow in the depthwise convolutionkernel (1) in FIG. 6, i.e. R*S) of the weight values in the depthwiseconvolution kernel, and two adjacent groups of points have a readingstride equal to a stride of the depthwise convolution calculation.

Depending on the reading stride, an overlapping portion may be locatedbetween every two adjacent groups of points in the p groups of points.

For example, at the first execution, the input feature values of theformer p groups of points on the former m input channels may be readfrom the input feature map (the input feature values shown in the inputfeature map (1) in FIG. 6).

Substep 2: corresponding to the input feature values of the firstpredetermined number p groups of points, reading corresponding weightvalues of the second predetermined number m of depthwise convolutionkernels on the second predetermined number m of input channels. In anexample, the input feature values of a first predetermined number p ofgroups of points on a second predetermined number m of input channelsand the corresponding weight values are read from the input feature map,concurrent with the step of performing pointwise convolutioncalculations according to the intermediate feature values of the firstpredetermined number p of points on the second predetermined number m ofdepthwise convolution output channels and the pointwise convolutionkernels. Therefore, based on this example, the computational efficiencyof the convolutional neural network is improved and a more efficientconvolutional neural network is achieved.

For example, at the first execution, the weight values (as the weightvalues in the depthwise convolution kernel (1) shown in FIG. 6) in theformer m depthwise convolution kernels may be read from the depthwiseconvolution kernels (as M depthwise convolution kernels (1) shown inFIG. 6).

Substep 3: respectively performing the depthwise convolutioncalculations on the input feature values of the first predeterminednumber p groups of points on the second predetermined number m of inputchannels and on weight values in corresponding second predeterminednumber m of depthwise convolution kernels, to obtain intermediatefeature values of the first predetermined number p of pointsrespectively corresponding to the first predetermined number p groups ofpoints on the second predetermined number m of depthwise convolutionoutput channels.

For example, the intermediate feature map (1) may be calculatedaccording to the input feature map (1) and the depthwise convolutionkernels (1).

For example, in substep 3, the following operations may be performed foreach group of points in the first predetermined number p groups ofpoints: (1) respectively performing multiplication calculations on inputfeature values of one point of the group of points on the secondpredetermined number m of input channels and one corresponding weightvalue in the corresponding second predetermined number m of depthwiseconvolution kernels, to obtain current multiplication calculationresults of the group of points; (2) respectively performing accumulationcalculations on the current multiplication calculation results of thegroup of points and the multiplication calculation results of theprevious time of the group of points, the multiplication calculationresults of the previous time being obtained by respectively performingmultiplication calculations on input feature values of previous onepoint in the group of points on the second predetermined number m ofinput channels and a corresponding previous weight value in thecorresponding second predetermined number m of depthwise convolutionkernels; and (3) repeating above operations (1) and (2), respectivelyperforming multiplication calculations on input feature values of a nextpoint of the group of points on the second predetermined number m ofinput channels and a corresponding next weight value in thecorresponding second predetermined number m of depthwise convolutionkernels and correspondingly performing subsequent operations, until themultiplication calculations and accumulation calculations are completedon the input feature values of all points in the group of points on thesecond predetermined number m of input channels, and the finalaccumulation calculation results of the group of points being anintermediate feature value of one point (as shown in the shadow inintermediate feature map (1) in FIG. 6) corresponding to the group ofpoints on the second predetermined number m of depthwise convolutionoutput channels.

After performing multiplication calculations on the first point in thegroup of points, since the multiplication calculation results of theprevious time of the group of points do not exist or are 0, therefore,there is no need to perform accumulation, in other words, theaccumulation calculation results are the multiplication calculationresults themselves.

As shown in FIG. 6, there are a total of R*S points in one group ofpoints, i.e. in substep (3), R*S multiplication calculations andcorresponding accumulation calculations are performed for a group ofpoints.

Through the above calculations, the intermediate feature values (theintermediate feature values shown in the intermediate feature map (1) inFIG. 6) of the first predetermined number p of points on the secondpredetermined number m of depthwise convolution output channels may beobtained.

For example, the intermediate feature values of the first predeterminednumber p of points on the second predetermined number m of depthwiseconvolution output channels may be stored in the memory after step S211.In other words, after obtaining intermediate feature values of the firstpredetermined number p of points on the second predetermined number m ofdepthwise convolution output channels, each intermediate feature valueis stored in the available space of the memory.

According to the current design parameters of the convolutional layer,at least one of the following operations may be performed for eachintermediate feature value after obtaining but before storing eachintermediate feature value: activation operation and quantizationoperation.

If the activation function is not added to the neural network, it may beregarded as linear expression to a certain extent, and the finalexpression ability is not good, and if some nonlinear activationfunctions are added, a nonlinear portion is introduced in the wholenetwork, and the expression ability of the network is increased. Atpresent, the popular activation functions mainly include Sigmoid, Tanh,ReLu, Softmax and so on.

Further, the quantization operation and inverse quantization operationmay also be introduced to the calculation data. For example,high-precision output data may be compressed into low precision outputdata by shifting or multiplication and division, such that the storagespace occupied by each data in the memory is reduced and the accessspeed is fully improved.

For example, the unquantized high precision data may be 32 bits, whilethe quantized low precision data may be 8 bits, such that 75% storagespace is saved.

For example, the optional activation operation may be performed, andthen the optional quantization operation is performed.

In step S212, the above operations are repeated (i.e. step S211), andthe depthwise convolution calculations are performed according to theinput feature maps and the depthwise convolution kernels, to obtain theintermediate feature values of the first predetermined number p ofpoints on a next second predetermined number m of depthwise convolutionoutput channels, and correspondingly performing subsequent operations,until the intermediate feature values of the first predetermined numberp of points on all depthwise convolutional output channels are obtained.

For example, next, as shown in FIG. 6, the intermediate feature map (2)may be calculated according to the input feature map (2) and thedepthwise convolution kernel (2), and so on, the intermediate featuremap (z) may be calculated according to the input feature map (z) and thedepthwise convolution kernel (z). For example, z=[M/m]. If, during thelast calculation, the number of remaining channels and the number ofdepthwise convolution kernels are not enough to read the input featurevalue of m channels with the weight values in the m depthwiseconvolution kernels, it may be realized through a padding (for example,padding 0) manner. Alternatively, it may also be realized by othermanners, for example reducing working depthwise convolution calculationunits.

Finally, for the combination from the intermediate feature map (1) tothe intermediate feature map (z), which means the complete intermediatefeature map including the intermediate feature values as theintermediate output results of the depthwise convolution operation, thesize of the intermediate feature map is p points and have a total of Mchannels, as shown in the intermediate feature map in FIG. 6.

Next, it will refer to FIGS. 7 and 8 to describe a step S220 ofperforming a pointwise convolution calculation on the intermediatefeature map to obtain output feature values of the first predeterminednumber p of points on all pointwise convolution output channels.

FIG. 7 shows a flowchart of the step of calculating the output featurevalues of the first predetermined number p of points on all pointwiseconvolution output channels according to the intermediate feature valuesand the pointwise convolution kernels in accordance with the firstembodiment of the present disclosure. FIG. 8 shows a schematic diagramof the step of calculating the output feature values of the firstpredetermined number p of points on all pointwise convolution outputchannels according to the intermediate feature values and the pointwiseconvolution kernels in accordance with the first embodiment of thepresent disclosure.

As shown in FIG. 7, the step S220 of calculating the output featurevalues of the first predetermined number p of points on all pointwiseconvolution output channels according to the intermediate feature valuesand the pointwise convolution kernels may comprise:

In step S221, a pointwise convolution calculation is performed accordingto the intermediate feature values of the first predetermined number pof points on all depthwise convolution output channels with the weightvalues on all pointwise convolution channels in a fourth predeterminednumber n of pointwise convolution kernels, respectively, to obtain theoutput feature values of the first predetermined number p of points onthe fourth predetermined number n of pointwise convolution outputchannels corresponding to the fourth predetermined number n of pointwiseconvolution kernels.

For example, the fourth predetermined number n is less than or equal tothe total number N of pointwise convolution kernels. Furthermore, thefourth predetermined number n may also be limited by the number ofpointwise convolution calculation units (e.g., multiplier-adder unitMAC), which may be a divisor of the number of multiplier-adder units.

For example, as shown in FIG. 8, the output feature map (1) may befirstly calculated based on the intermediate feature map and thepointwise convolution kernel (1). In FIG. 8, the pointwise convolutionkernel (1) is a set of n pointwise convolution kernels.

In an example, the step S221 may comprise:

In an example, the step S221 may comprise:

Substep 1: reading intermediate feature values (as intermediate featurevalues shown in the intermediate feature map (1) in FIG. 8) of the firstpredetermined number p of points on a third predetermined number m′ ofdepthwise convolution output channels from intermediate feature valuesof the first predetermined number p of points on all depthwiseconvolution output channels (as shown in the intermediate feature map(1) in FIG. 8).

For example, the third predetermined number m′ and the fourth presetnumber n may be determined according to the number of pointwiseconvolution calculation units and the first predetermined number p. Forexample, based on computational efficiency considerations, it isdesirable to make the pointwise convolution calculation units in thehardware circuit operate at full capacity. In this case, the thirdpredetermined number m′*the fourth predetermined number n*the firstpredetermined number p*=the number of pointwise convolution calculationunit MAC′.

For example, for different hardware designs, the number of pointwiseconvolution calculation units MAC′ may or may not be equal to the numberof depthwise convolution calculation unit MAC. Further, the thirdpredetermined number m′ may be less than or equal to the secondpredetermined number n.

Assuming that there are 512 multiplier-adder units for pointwiseconvolution calculation too, for example, the pointwise convolutioncalculation of 32 points (p=32), 4 channels (m′=4), and 4 convolutionkernels (n=4) may be simultaneously calculated at a time. As mentionedabove, different values of m and n may also be selected for otherconsiderations. For example, when the number of current convolutionkernels is relatively large and the number of channels is relativelysmall, in a case where p=32 remains unchanged, n may be set to a largervalue, for example, 8, 16, etc., correspondingly, m′ may be set to asmaller value, for example, 2, 1, etc. Conversely, when the number ofcurrent convolution kernels is relatively small and the number ofchannels is relatively large, in a case where p=32 remains unchanged, nmay be set to a smaller value, for example, 2, 1, etc., correspondingly,m′ may be set to a smaller value, for example, 8, 16, etc.

For example, at the first execution, the intermediate feature values ofp points on the former m′ channels may be read from the intermediatefeature map obtained by the depthwise convolution calculation, as shownin the intermediate feature map (1) in FIG. 8.

It should be noted that, in the reading operation in the substep, theintermediate feature values of fifth predetermined number p′ points onthe third predetermined number m′ of depthwise convolution outputchannels may also be read. For example, the fifth predetermined numberp′ is less than or equal to the first predetermined number p.

Substep 2: reading the weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in the fourthpredetermined number n of pointwise convolution kernels correspondinglyto the intermediate feature values of the first predetermined number pof points on the third predetermined number m′ of depthwise convolutionoutput channels.

For example, at the first execution, the weight values on the former m′pointwise convolution channels (such as the weight values shown by thepointwise convolution kernel (11) in the pointwise convolution kernel(1) in FIG. 8) may be read from former n pointwise convolution kernelsin N pointwise convolution kernels (as shown in the pointwiseconvolution kernel (1) in FIG. 8).

Substep 3: performing pointwise convolution calculation on theintermediate feature values of the first predetermined number p ofpoints on the third predetermined number m′ of depthwise convolutionoutput channels with the weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in the fourthpredetermined number n of pointwise convolution kernels, to obtain thecurrent pointwise convolution partial sums of the first predeterminednumber p of points on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels.

For example, the current pointwise convolution partial sums (1) may becalculated according to the intermediate feature map (1) and thepointwise convolution kernel (11) (as shown in the output feature map(11) in FIG. 8).

For example, in substep 3, following operations are performed for eachpoint in the first predetermined number p of points: (1) respectivelyperforming multiplication calculations on the intermediate featurevalues of the first predetermined number p of points on the thirdpredetermined number m′ of depthwise convolution output channels withthe weight values on the corresponding third predetermined number m′ ofpointwise convolution channels in the fourth predetermined number n ofpointwise convolution kernels, to obtain results of a fourthpredetermined number n of groups, results of each group including athird predetermined number m′ of multiplication calculation results; and(2) respectively adding the third predetermined number m′ ofmultiplication calculation results for results of each group in resultsof the fourth predetermined number n of groups, to obtain the currentpointwise convolution partial sums of this point on the fourthpredetermined number n of pointwise convolution output channelscorresponding to the fourth predetermined number n of pointwiseconvolution kernels.

As shown in FIG. 8, for each of p points in the intermediate feature map(1), multiplication calculations are respectively performed on the m′intermediate feature values of each point and the m′ weight values in afirst convolution kernel of the pointwise convolution kernels (11), andthe m′ multiplication calculation results are accumulated, to obtain thecurrent pointwise convolution partial sums of p points in the firstoutput feature map (11) on the first channel. The above operations areperformed in each convolution kernel of the pointwise convolutionkernels (11) to obtain the current pointwise convolution partial sums ofp points in the first output feature map (11) on the former n channel.

Substep 4: respectively performing accumulation calculations on thecurrent pointwise convolution partial sums of the first predeterminednumber p of points on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels and the previous accumulationcalculation results of the first predetermined number p of points, togenerate current accumulation calculation results of the firstpredetermined number p of points.

After obtaining the current pointwise convolution partial sums of ppoints in the first output feature map (11) on the former n channel,since the previous accumulation calculation results of this p points donot exist or are 0, therefore, there is no need to perform accumulation,or say, the accumulation calculation results are the multiplicationcalculation results themselves.

For example, after substep 4, the current accumulation calculationresults of the first predetermined number p of points may be stored inthe memory to cover the previous accumulation calculation results of thefirst predetermined number p of points.

Substep 5: repeating above operations (i.e. substeps 1-4), reading theintermediate feature values of the first predetermined number p ofpoints on a next third predetermined number m′ of depthwise convolutionoutput channels, reading weight values on a corresponding next thirdpredetermined number m′ of pointwise convolution channels in a fourthpredetermined number n of pointwise convolution kernels, andcorrespondingly performing subsequent operations until the pointwiseconvolution calculations and accumulation calculations are completed onthe intermediate feature values of the first predetermined number p ofpoints on all depthwise convolution output channels, the finalaccumulation calculation results of the first predetermined number p ofpoints being the output feature values of the first predetermined numberp of points on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels.

For example, next, as shown in FIG. 8, the output feature map (12) maybe calculated according to the intermediate feature map (2) and thepointwise convolution kernel (12), and an accumulation calculation maybe performed on the output feature map (12) and the output feature map(11), and so on, the output feature map (1z′) is calculated according tothe intermediate feature map (z′) and the pointwise convolution kernel(1z′), and the accumulation calculation is performed on the outputfeature map (1z′) and the output feature map (1(z′−1)). For example,z′=[M/m′]. If, during the last calculation, the number of remainingchannels is not enough to read the intermediate feature values of m′channels and pointwise convolution weight values of m′ channels, it maybe realized through padding (for example, padding 0) manner.Alternatively, it may also be realized by other manners, for examplereducing working pointwise convolution calculation units.

Finally, the accumulation results of the output feature map (1z′) andthe output feature map (1(z′−1)) are made as the output feature valuesof p points on n pointwise convolution output channels, as shown in theoutput feature map (1) in FIG. 8, the size of output feature map (1) isp points, with a total of N channels.

For example, at least one of an activation operation and a quantizationoperation may be performed on each output feature value before the finalaccumulation calculation results of the first predetermined number p ofpoints are stored in the memory as output feature values of the firstpredetermined number p of points on the fourth predetermined number n ofpointwise convolution output channels corresponding to the fourthpredetermined number n of pointwise convolution kernels.

In step S222, the above operations are repeated (i.e. step S211), thepointwise convolution calculations are performed according to theintermediate feature values of the first predetermined number p ofpoints on all depthwise convolution output channels with the weightvalues on all pointwise convolution channels in a next fourthpredetermined number n of pointwise convolution kernels, respectively,until the output feature values of the first predetermined number p ofpoints on all pointwise convolution output channels are obtained.

For example, next, as shown in FIG. 8, the output feature map (2) may becalculated according to the intermediate feature map and the pointwiseconvolution kernel (2), and so on, the output feature map (g) may becalculated according to the intermediate feature map and the pointwiseconvolution kernel (g). For example, g=[N/n]. If during the lastcalculation, the number of remaining pointwise convolution kernels isnot enough to read the weight values in the n pointwise convolutionkernels, it may also be realized through padding (for example, padding0) manner. Alternatively, it may also be realized by other manners, forexample, reducing working pointwise convolution calculation units.

Finally, for the combination from the output feature map (1) to theoutput feature map (g), which means the complete output feature mapincluding the output feature values as the final output result of thepointwise convolution operation, as shown in the output feature diagramin FIG. 8, the size of the output feature map is p points and have atotal of N channels.

As described above, the output feature map may perform the optionalactivation operation and the optional quantization operation.

Hereinafter, a method for convolution calculation according to the firstembodiment of the present disclosure will be explained in a specificexample.

For example, as shown in FIG. 6 and FIG. 8, assuming that the width andheight dimensions of the depthwise convolution are S*R, the number ofchannels (or may also be seen as number when the channel number isconsidered to be 1) is M; the width and height dimensions of thedepthwise convolution are 1*1, the number of channels is M, and thenumber is N; the width and height dimensions of the input feature mapare X*Y, the width and height dimensions of the intermediate feature mapare W*H, and the width and height dimensions of the output feature mapare E*F, the convolution calculation method may comprise the followingsteps:

1. For depthwise convolution, firstly calculating multiplication andaccumulation results of p(p<=H*W) points and m(m<=M) channels, theaccumulation here being the accumulation performed in the direction ofthe length and width of the convolution kernel, as R and S shown in FIG.6, and here p*m multiply-accumulate (MAC) units being shared, and p*mmultiply-accumulate results being obtained.

FIG. 9 shows a schematic diagram of a depthwise convolution calculationaccording to an embodiment of the present disclosure.

As shown in FIG. 9, it is assumed that there are 512 multiplier-adderunits (MAC), and 32 points (the abovementioned p=32), 16 channels (theabovementioned m=16, i.e. the channel variable c takes a value from therange of 0 to 15) are simultaneously calculated at a time. Assuming thatthe size of the convolution kernel is 3*3, then, after 9 times ofmultiplication and accumulation calculations (Mult and Accu) (the heightand width variables r and s of the depthwise convolution kernel changesfrom 0 to 2, respectively, the width variable x of the input feature mapchanges from 0-31 to 2-33, and the height variable y changes from 0 to2), the output values of 32 points, 16 channels will be obtained.Assuming that there are a total of 128 channels, after (128/16)*9=72times of calculations, the output data of 32 points, 128 channels willbe obtained.

2. Performing an optional activation operation on the abovementionedresults of step 1, the activation operation referring to remapping thenumerical with a nonlinear function, the activation functions includingbut not limited to ReLu function, Sigmoid function, arctangent (tan)function, etc.

3. Performing an optional quantization operation on results obtained bythe abovementioned step 2, the quantization operation referring toobtain the low-precision multiplication and accumulation results(usually 8 bit) by shifting or multiplying and dividing thehigh-precision multiplication and accumulation results (usually 32 bit).

4. Storing the abovementioned results of step 3 in the register or theon-chip SRAM.

5. Through [M/m] times of circulation, obtaining p*M values aftercalculating the depthwise convolution results of p points on the Mchannels, the p*M values being or being not carried out the activationfunction operation and the quantization operation. Since the number ofpoints of a layer of complete output results is much greater than p,assuming that there is a total of H*W points, the storage space usedhere being only p/(H*W) for calculating a complete layer of depthwiseoperation results.

6. Directly performing pointwise calculation on the abovementionedresults of step 5. The specific process is as follows:

a). Reading the results of depthwise convolution calculation of ppoints, m channels from the register or the on-chip SRAM, andcalculating the multiplication and accumulation results of p points, mchannels, the accumulation here being the accumulation in the channeldirection, to obtain the pointwise convolution partial sums of p points,1 output channel (the partial sums of the first to m-th input channels).

b). Obtaining the final multiplication and accumulation results of ppoints after calculation on the pointwise convolution results of ppoints through [M/m] times of circulation.

c). Performing an optional activation operation on the above results,the activation operation referring to remapping the numerical with anonlinear function, and the activation functions including but notlimited to ReLu function, Sigmoid function, arctangent (tan) function,etc.

d). Performing an optional quantization operation on the above results,the quantization operation referring to obtaining the low-precisionmultiplication and accumulation results (usually 8 bit) by shifting ormultiplying and dividing the high-precision multiplication andaccumulation results (usually 32 bit), and storing the results in theregister or the on-chip SRAM or the off-chip DDR.

e). Completing output of the calculation results of p points on Nchannels through N circulation of the above operations a), b), c), d),assuming that there is a total of N convolution kernels for pointwiseconvolution.

FIG. 10 shows a schematic diagram of a pointwise convolution calculationaccording to an embodiment of the present disclosure.

As shown in FIG. 10, the size of a convolution kernel of the Pointwiseconvolution is 1*1, and it is also assumed that there are 512multiplier-adder units (the 512 multiplier-adder units here and the 512multiplier-adder units of the Depthwise calculation process may be thesame, or may be different), and a convolution for 32 points (theabovementioned p=32), 16 channels (the abovementioned m=16, i.e. thechannel variable c takes a value from the range of 0 to 15), and 1convolution kernel are calculated each time, and part of data of theaccumulation sums for 32 points, 1 output channel are obtained. Assumingthat there is a total of 128 channels, after 128/16=8 times ofcalculations (the channel variable c changes from 0-15 to 112-127), thedata of the accumulation sums of 32 points and 1 output channel areobtained. And assuming that there are 256 convolution kernels, thenafter 256*(128/16)=2048 times of calculations, the data of theaccumulation sums of 32 points and 256 output channels are obtained.That is, an output feature value of 32 points is obtained.

7. Repeating the above operations of steps 1-6 by continuouslycalculating next p points until the complete output feature map isobtained.

Second Embodiment

In the second embodiment of the present disclosure, in order to obtainthe output feature values of the first predetermined number p of pointson all pointwise convolution output channels, the following operationsmay be performed: 1) performing depthwise convolution operations toobtain the intermediate feature values of the first predetermined numberp of points on all depthwise convolution output channels; (2) performingpointwise convolution operations according to the intermediate featurevalues to obtain a current pointwise convolution partial sums for ppoints on all of the pointwise convolution output channels; (3)performing accumulation on the current pointwise convolution partialsums and the previous accumulation calculation results, to generate thecurrent accumulation calculation results; (4) repeating above operations(2) and (3) to obtain the output feature values of p points on allpointwise convolution output channels.

That is, the performing depthwise convolution operation in the secondembodiment (step S210) is the same as the performing depthwiseconvolution operation in the first embodiment, and the performingpointwise convolution operation (step S220) in the second embodiment isdifferent from the performing pointwise convolution operation in thefirst embodiment. Therefore, next, the differences between the twoembodiments will be emphatically described.

FIG. 11 shows a flowchart of the step of calculating the output featurevalues of the first predetermined number p of points on all pointwiseconvolution output channels according to the intermediate feature valuesand the pointwise convolution kernels according to the second embodimentof the present disclosure. FIG. 12 shows a schematic diagram of the stepof calculating the output feature values of the first predeterminednumber p of points on all pointwise convolution output channelsaccording to the intermediate feature values and the pointwiseconvolution kernels according to the second embodiment of the presentdisclosure.

As shown in FIG. 11, the step S220 of calculating the output featurevalues of the first predetermined number p of points on all pointwiseconvolution output channels according to the intermediate feature valuesand the pointwise convolution kernels may comprise:

In step S223, performing the pointwise convolution calculationsaccording to intermediate feature values of the first predeterminednumber p of points on the third predetermined number m′ of depthwiseconvolution output channels and weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in allpointwise convolution kernels, respectively, to obtain the currentpointwise convolution partial sums of the first predetermined number pof points on all pointwise convolution output channels.

For example, the third predetermined number m′ is less than or equal tothe second predetermined number m. Furthermore, the third predeterminednumber m′ may also be limited by the number of pointwise convolutioncalculation units (e.g., multiplier-adder unit MAC), which may be adivisor of the number of the multiplier-adder units.

In an example, the step S223 may comprise:

Substep 1: performing the pointwise convolution calculations accordingto the intermediate feature values of the first predetermined number pof points on the third predetermined number m′ of depthwise convolutionoutput channels with the weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in the fourthpredetermined number n of pointwise convolution kernels, to obtain thecurrent pointwise convolution partial sums of the first predeterminednumber p of points on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels.

For example, in substep 1, the following steps may be performed:

(1) Reading the intermediate feature values of the first predeterminednumber p of points on the third predetermined number m′ of depthwiseconvolution output channels from the intermediate feature values of thefirst predetermined number p of points on all depthwise convolutionoutput channels.

For example, at the first execution, the intermediate feature values ofp points on the former m′ channels may be read from the intermediatefeature map obtained by the depthwise convolution calculations, as shownin the intermediate feature map (1) in FIG. 12.

(2) Reading the weight values on the corresponding third predeterminednumber m′ of pointwise convolution channels in the fourth predeterminednumber n of pointwise convolution kernels correspondingly to the inputfeature values of the first predetermined number p of points on thethird predetermined number m′ of depthwise convolution output channels.

For example, at the first execution, the weight values on the former m′pointwise convolution channels (such as the weight values shown by thepointwise convolution kernel (1) of the pointwise convolution kernel (1)in FIG. 12) may be read from the former n pointwise convolution kernelsin N pointwise convolution kernels (as shown in the pointwiseconvolution kernel (1) in FIG. 12).

(3) Performing the pointwise convolution calculations on intermediatefeature values of the first predetermined number p of points on thethird predetermined number m′ of depthwise convolution output channelsand weight values on a corresponding third predetermined number m′ ofpointwise convolution channels in the fourth predetermined number n ofpointwise convolution kernels, to obtain the current pointwiseconvolution partial sums of the first predetermined number p of pointson the fourth predetermined number n of pointwise convolution outputchannels corresponding to the fourth predetermined number n of pointwiseconvolution kernels.

For example, the current pointwise convolution partial sums (11) may becalculated based on the intermediate feature map (1) and the pointwiseconvolution kernel (11) (as shown in the output feature map (11) in FIG.12).

This step is the same as substep 3 of step S221 in the first embodiment,therefore, a detailed description thereof will be omitted.

Substep 2: repeating the above operations by performing the pointwiseconvolution calculations on the intermediate feature values of the firstpredetermined number p of points on the third predetermined number m′ ofdepthwise convolution output channels and weight values on thecorresponding third predetermined number m′ of pointwise convolutionchannels in a next fourth predetermined number n of pointwiseconvolution kernels, to obtain a current pointwise convolution partialsums of the first predetermined number p of points on the next fourthpredetermined number n of pointwise convolution output channelscorresponding to the next fourth predetermined number n of pointwiseconvolution kernels, until the current pointwise convolution partialsums of the first predetermined number p of points on all pointwiseconvolutional output channels are obtained.

For example, next, as shown in FIG. 12, the current pointwiseconvolution partial sums (21) may be calculated according to theintermediate feature map (1) and the pointwise convolution kernel (21)(as shown in the output feature map (21) in FIG. 12), and so on, thecurrent pointwise convolution partial sums (g1) are calculated accordingto the intermediate feature map (1) and the pointwise convolution kernel(g1) (as shown in the output feature map (g1) in FIG. 12).

Finally, the current pointwise convolution partial sums (11) to thecurrent pointwise convolution partial sums (g1) are combined, whichmeans the current pointwise convolution partial sums of p points on allof the pointwise convolution output channels (1) of p points on allpointwise convolution output channels, the size of which are p pointsand have a total of N channels.

In step S224, accumulation calculations are respectively performed onthe current pointwise convolution partial sums of the firstpredetermined number p of points on all pointwise convolution outputchannels and previous accumulation calculation results of the firstpredetermined number p of points, to generate current accumulationcalculation results of the first predetermined number p of points.

After obtaining the current pointwise convolution partial sums (1) of ppoints on all of the pointwise convolution output channels, since theprevious accumulation calculation results do not exist or are 0,therefore, there is no need to perform accumulation, or say, the currentpointwise convolution partial sums (1) are the current accumulationcalculation results.

After the step S224, the current accumulation calculation results arestored in the memory to cover the previous accumulation calculationresults of the first predetermined number p of points.

In step S225, the above operations (i.e. step S223 and step S224) arerepeated, pointwise convolution calculations are performed according tointermediate feature values of the first predetermined number p ofpoints on the next third predetermined number m′ of depthwiseconvolution output channels and weight values on the corresponding nextthird predetermined number m′ of pointwise convolution channels in allpointwise convolution kernels, respectively, and subsequent operationsare correspondingly performed, until the pointwise convolutioncalculations and accumulation calculations are completed on all of theintermediate feature values of the first predetermined number p ofpoints on all depthwise convolution output channels, the finalaccumulation calculation results of the first predetermined number p ofpoints being the output feature values of the first predetermined numberp of points on all pointwise convolution output channels.

For example, next, as shown in FIG. 12, the current pointwiseconvolution partial sums (12) may be calculated according to theintermediate feature map (2) and the pointwise convolution kernel (12),and the current pointwise convolution partial sums (22) may becalculated according to the intermediate feature map (2) and thepointwise convolution kernel (22), and so on, the current pointwiseconvolution partial sums (g2) are calculated according to theintermediate feature map (2) and the pointwise convolution kernel (g2).

The current pointwise convolution partial sums (12) to the currentpointwise convolution partial sums (g2) are combined together to obtainthe current pointwise convolution partial sums (2) of p points on all ofthe pointwise convolution output channels, the size of which are ppoints have a total of N channels.

After obtaining the current pointwise convolution partial sums (2) of ppoints on all of the pointwise convolution output channels, the currentpointwise convolution partial sums (2) are accumulated with the previousaccumulation calculation results (i.e. the current pointwise convolutionpartial sums (1)).

And so on, the current pointwise convolution partial sums (1z′) to thecurrent pointwise convolution partial sums (gz′) are combined togetherto obtain the current pointwise convolution partial sums (z′) of ppoints on all of the pointwise convolution output channels, the size ofwhich are p points and have a total of N channels.

After obtaining the current pointwise convolution partial sums (z′) of ppoints on all of the pointwise convolution output channels, the currentpointwise convolution partial sums (z′) are accumulated with theprevious accumulation calculation results (i.e. the current pointwiseconvolution partial sums (1) to the current pointwise convolutionpartial sums (z′-1)), such that an output feature map including theoutput feature values as the final output results of the pointwiseconvolution operation is obtained, as shown in the output featurediagram in FIG. 8, the size of which are p points and have a total of Nchannels.

For example, at least one of an activation operation and a quantizationoperation may be performed on each output feature value before the finalaccumulation calculation results of the first predetermined number p ofpoints is stored in the memory as the output feature values of the firstpredetermined number p of points on all pointwise convolution outputchannels.

Third Embodiment

In the third embodiment of the present disclosure, in order to obtainoutput feature values of a first predetermined number p of points on allpointwise convolution output channels, the following operations may beperformed: (1) performing depthwise convolution operations to obtain theintermediate feature values of the first predetermined number p ofpoints on a second predetermined number m of depthwise convolutionoutput channels; (2) performing pointwise convolution operationsaccording to the intermediate feature values to obtain the currentpointwise convolution partial sums of p points on all of the pointwiseconvolution output channels; (3) performing accumulation on the currentpointwise convolution partial sums and the previous accumulationcalculation results, to generate the current accumulation calculationresults; (4) repeating the above operations (1) and (3) to obtain theoutput feature values of p points on all pointwise convolution outputchannels.

FIG. 13 shows a flowchart of the step of calculating the output featurevalues of the first predetermined number p of points on all pointwiseconvolution output channels according to the third embodiment of thepresent disclosure. FIG. 14 shows a schematic diagram of the step ofcalculating the current pointwise convolution partial sums (i,1≤i≤z,z=[M/m]) of the first predetermined number p of points on all pointwiseconvolution output channels according to the third embodiment of thepresent disclosure.

As shown in FIG. 13, according to the third embodiment of the presentdisclosure, the step S110 of calculating the output feature values ofthe first predetermined number p of points on all pointwise convolutionoutput channels may comprise:

In step S310, depthwise convolution calculations are performed accordingto the input feature map and the depthwise convolution kernels to obtainthe intermediate feature values of the first predetermined number p ofpoints on the second predetermined number m of depthwise convolutionoutput channels.

For example, the second predetermined number m is determined accordingto the number of depthwise convolution calculation units and the firstpredetermined number p.

In an example, the step S310 may comprise:

Substep 1: reading input feature values of the first predeterminednumber p group of points on the second predetermined number m of inputchannels from the input feature map.

Substep 2: reading weight values in the corresponding secondpredetermined number m of depthwise convolution kernels correspondinglyto the input feature values of the first predetermined number p group ofpoints on the second predetermined number m of input channels.

Substep 3: respectively performing the depthwise convolutioncalculations on the input feature values of the first predeterminednumber p group of points on the second predetermined number m of inputchannels with the weight values in the corresponding secondpredetermined number m of depthwise convolution kernels, to obtainintermediate feature values of the first predetermined number p ofpoints respectively corresponding to the first predetermined number pgroups of points on the second predetermined number m of depthwiseconvolution output channels.

The substeps 1-3 in this step S310 are the same as the substeps 1-3 ofthe step S211 in the first embodiment, therefore, a detailed descriptionthereof will be omitted.

For example, at the first execution, firstly, the input feature valuesof the former p group of points on the former m input channels may beread from the input feature map (as the input feature values shown inthe input feature map (i=1) in FIG. 14). Then, the weight values (as theweight values shown in the depthwise convolution kernel (1) in FIG. 6)in the former m depthwise convolution kernels may be read from thedepthwise convolution kernels (as shown by the M depthwise convolutionkernels in FIG. 6). Finally, the intermediate feature map (i=1) may becalculated according to the input feature map (1) and the depthwiseconvolution kernel (1).

For example, the intermediate feature values of the first predeterminednumber p of points on the second predetermined number m of depthwiseconvolution output channels may be stored in the memory after substep 3.In other words, after obtaining the intermediate feature values of thefirst predetermined number p of points on the second predeterminednumber m of depthwise convolution output channels, each intermediatefeature value is stored in the available space of the memory.

Further, according to the current design parameters of the convolutionallayer, at least one of the following operations may be performed foreach intermediate feature value after each intermediate feature value isobtained and before it is stored: an activation operation and aquantization operation.

In step S320, pointwise convolution calculations are performed accordingto the intermediate feature values of the first predetermined number pof points on the second predetermined number m of depthwise convolutionoutput channels and the pointwise convolution kernels, to obtain thecurrent pointwise convolution partial sums of the first predeterminednumber p of points on all the pointwise convolution output channels.

In an example, the step S320 may comprise:

Substep 1: performing the pointwise convolution calculations accordingto intermediate feature values of the first predetermined number p ofpoints on the third predetermined number m′ of depthwise convolutionoutput channels and weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in allpointwise convolution kernels, respectively, to obtain the currentpointwise convolution sub-partial sums of the first predetermined numberp of points on all pointwise convolution output channels.

For example, in substep 1, the following steps may be performed: (1)performing the pointwise convolution calculations according to theintermediate feature values of the first predetermined number p ofpoints on the third predetermined number m′ of depthwise convolutionoutput channels and weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in a fourthpredetermined number n of pointwise convolution kernels, to obtain thecurrent pointwise convolution sub-partial sums of the firstpredetermined number p of points on the fourth predetermined number n ofpointwise convolution output channels corresponding to the fourthpredetermined number n of pointwise convolution kernels.

For example, the third predetermined number m′ and the fourthpredetermined number n may be determined according to the number ofpointwise convolution calculation units and the first predeterminednumber p.

For example, the operation (1) may comprise: (1-1) reading theintermediate feature values of the first predetermined number p ofpoints on the third predetermined number m′ of depthwise convolutionoutput channels from the intermediate feature values of the firstpredetermined number p of points on the second predetermined number m ofdepthwise convolution output channels; (1-2) reading the weight valueson the corresponding third predetermined number m′ of pointwiseconvolution channels in the fourth predetermined number n of pointwiseconvolution kernels correspondingly to the intermediate feature valuesof the first predetermined number p of points on the third predeterminednumber m′ of depthwise convolution output channels; (1-3) respectivelyperforming the pointwise convolution calculations on the intermediatefeature values of the first predetermined number p of points on thethird predetermined number m′ of depthwise convolution output channelswith the weight values on the corresponding third predetermined numberm′ of pointwise convolution channels in the fourth predetermined numbern of pointwise convolution kernels, to obtain the current pointwiseconvolution sub-partial sums of the first predetermined number p ofpoints on the fourth predetermined number n of pointwise convolutionoutput channels corresponding to the fourth predetermined number n ofpointwise convolution kernels.

Specifically, in (1-3), the following operations may be performed foreach of the first predetermined number p of points: performingmultiplication calculations on the intermediate feature values of thepoint on the third predetermined number m′ of depthwise convolutionoutput channels with the weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in the fourthpredetermined number n of pointwise convolution kernels, respectively,to obtain the results of a fourth predetermined number n of groups, eachgroup of which comprise a third predetermined number m′ ofmultiplication calculation results; and respectively adding the thirdpredetermined number m′ of multiplication calculation results in theresults of each group of the results of the fourth predetermined numbern of groups, to obtain the current pointwise convolution sub-partialsums for this point on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels.

(2) Repeating the above operations by performing the pointwiseconvolution calculations on the intermediate feature values of the firstpredetermined number p of points on the third predetermined number m′ ofdepthwise convolution output channels with the weight values on thecorresponding third predetermined number m′ of pointwise convolutionchannels in the next fourth predetermined number n of pointwiseconvolution kernels, to obtain the current pointwise convolutionsub-partial sums of the first predetermined number p of points on thenext fourth predetermined number n of pointwise convolution outputchannels corresponding to the next fourth predetermined number n ofpointwise convolution kernels, until the current pointwise convolutionsub-partial sums of the first predetermined number p of points on allpointwise convolutional output channels are obtained.

Substep 2: respectively performing accumulation calculations on thecurrent pointwise convolution sub-partial sums of the firstpredetermined number p of points on all pointwise convolution outputchannels and the previous accumulation calculation sub-results of thefirst predetermined number p of points, to generate the currentaccumulation calculation sub-results of the first predetermined number pof points.

For example, after generating current accumulation calculationsub-results of the first predetermined number p of points, the currentaccumulation calculation sub-results may be stored in the memory tocover the previous accumulation calculation sub-results of the firstpredetermined number p of points.

Substep 3: repeating the above operations by performing the pointwiseconvolution calculations according to the intermediate feature values ofthe first predetermined number p of points on the next thirdpredetermined number m′ of depthwise convolution output channels withthe weight values on the corresponding next third predetermined numberm′ of pointwise convolution channels in all pointwise convolutionkernels, respectively, and correspondingly performing the subsequentoperation, until the pointwise convolution calculations and accumulationcalculations are completed on all of the intermediate feature values ofthe first predetermined number p of points on the second predeterminednumber m of depthwise convolution output channels, the finalaccumulation calculation sub-results of the first predetermined number pof points being the current pointwise convolution partial sums of thefirst predetermined number p of points on all pointwise convolutionoutput channels.

Substeps 1-3 of step S320 are substantially similar to steps S223-S225of step S220 in the second embodiment, and thus the detailed descriptionthereof is omitted.

For example, at the first execution, firstly, the intermediate featurevalues of p points on the former m′ depthwise convolution outputchannels may be read from the intermediate feature map obtained bydepthwise convolution calculations. Then, the weight values on theformer m′ pointwise convolution channels may be read from the former npointwise convolution kernels in the N pointwise convolution kernels.Next, the pointwise convolution sub-partial sums on the former npointwise convolution output channels may be calculated according to theboth. Then, the pointwise convolution sub-partial sums on the latter npointwise convolution output channels is calculated, until the pointwiseconvolution sub-partial sums on the N pointwise convolution outputchannels is obtained. Next, the sub-partial sums are accumulated, andthe intermediate feature values of p points on the latter m′ channelsare read, and the above operations are repeated to obtain the currentpointwise convolution partial sums of p points on all of the pointwiseconvolution output channels.

It should be noted that what is described herein is that iteration isfirstly performed on the dimension (n) of the number of the pointwiseconvolution kernels and then iteration is performed on the dimension (m)of the number of channels, however, embodiments of the presentdisclosure are not limited to it. For example, the iteration may also beperformed on the dimension (m) of the number of channels, and then theiteration is performed on the dimension (n) of the number of thepointwise convolution kernels.

In a simple case, if m′=m, then at the first execution, firstly, theintermediate feature values of p points on the first m channels may beread, as shown in the intermediate feature map (i=1) in FIG. 14. Then,the weight values on the former m pointwise convolution channels in theN pointwise convolution kernels may be read, as shown in the pointwiseconvolution kernel (i=1) in FIG. 14. Next, the current pointwiseconvolution partial sums may be calculated according to the both, asshown in the output feature map (i=1) in FIG. 14.

In step S330, accumulation calculations are respectively performed onthe current pointwise convolution partial sums of the firstpredetermined number p of points on all pointwise convolution outputchannels and the previous accumulation calculation results of the firstpredetermined number p of points, to generate the current accumulationcalculation results of the first predetermined number p of points.

For example, after generating the current accumulation calculationsub-results of the first predetermined number p of points, the currentaccumulation calculation results are stored in the memory to cover theprevious accumulation calculation results of the first predeterminednumber p of points.

In step S340, the above operations are repeated, and the pointwiseconvolution calculations are performed according to the intermediatefeature values of the first predetermined number p of points on the nextsecond predetermined number m of depthwise convolution output channelsand the pointwise convolution channels, and the subsequent operationsare correspondingly performed, until the pointwise convolutioncalculations and accumulation calculations are completed on all of theintermediate feature values of the first predetermined number p ofpoints on all depthwise convolution output channels, and the finalaccumulation calculation results of the first predetermined number p ofpoints being the output feature values of the first predetermined numberp of points on all pointwise convolution output channels.

For example, next, the intermediate feature map (i=2) may be calculatedaccording to the input feature map (i=2) and the depthwise convolutionkernel (i=2), then, the current pointwise convolution partial sums maybe calculated according to the intermediate feature map (i=2) and thepointwise convolution kernel (i=2), as shown in the output feature map(i=2) in FIG. 14, next, the output feature map (i=2) is accumulated withthe previous accumulation calculation results (i.e. the output featuremap (i=1) in FIG. 14) to generate the current accumulation calculationresults. In a similar fashion, the intermediate feature map (i=z) may becalculated according to the input feature map (i=z) and the depthwiseconvolution kernel (i=z), and then, the current pointwise convolutionpartial sums may be calculated according to the intermediate feature map(i=z) and the pointwise convolution kernel (i=z) as shown in the outputfeature map (i=z) in FIG. 14, next, the output feature map (i=z) isaccumulated with the previous accumulation calculation results (i.e. thecurrent pointwise convolution partial sums (1) to the current pointwiseconvolution partial sums (z′−1)), such that an output feature mapincluding the output feature values as the final output results of thepointwise convolution operation are obtained, as shown in the outputfeature diagram in FIG. 8, the size of which are p points and have atotal of N channels.

For example, at least one of an activation operation and a quantizationoperation may be performed on each output feature value before the finalaccumulation calculation results of the first predetermined number p ofpoints are stored in the memory as the output feature value of the firstpredetermined number p of points on all pointwise convolution outputchannels.

Hereinafter, the convolution calculation method according to the firstembodiment of the present disclosure will be explained in a specificexample.

For example, as shown in FIG. 14, assuming that the width and heightdimensions of the depthwise convolution are S*R, the number of channels(or may also be regarded as number when the channel number is consideredto be 1) is M, the width and height dimensions of the depthwiseconvolution are 1*1, the number of channels is M, and the number is N,the width and height dimensions of the input feature map are X*Y, thewidth and height dimensions of the intermediate feature map are W*H, andthe width and height dimensions of the output feature map are E*F, thenthe convolution calculation method may comprise the following steps:

1. For the depthwise convolution, firstly, calculating themultiplication and accumulation results of p(p<=H*W) points and m(m<=M)channels, the accumulation here being the accumulation performed in thedirection of the length and width of the convolution kernel, as R and Sshown in FIG. 6, and p*m multiply-accumulate (MAC) units being sharedhere, and p*m multiply-accumulate results being obtained.

As shown in FIG. 9, it is assumed that there are 512 multiplier-adderunits (MACs), and 32 points (the abovementioned p=³²), 16 channels (theabovementioned m=16, i.e. the channel variable c takes a value from therange of 0 to 15) are simultaneously calculated at a time. Assuming thatthe size of the convolution kernel is 3*3, then, after 9 times ofmultiplication and accumulation calculations (Mult and Accu) (the heightand width variables r and s of the depthwise convolution kernel changesfrom 0 to 2, respectively, the width variable x of the input feature mapchanges from 0-31 to 2-33, and the height variable y changes from 0 to2), the output values of 32 points, 16 channels will be obtained.

2. Performing an optional activation operation on the results ofabovementioned step 1, the activation operation referring to remappingthe numerical with a nonlinear function, the activation functionsincluding but not limited to ReLu function, Sigmoid function, arctangent(tan) function, etc.

3. Performing an optional quantization operation on the results of theabovementioned step 2, the quantization operation referring to obtainingthe multiplication and accumulation results having low precision(usually 8 bit) by shifting or multiplying and dividing themultiplication and accumulation results having high precision (usually32 bit).

4. Storing the results of abovementioned step 3 in a register or on-chipSRAM.

5. Directly performing pointwise calculations for the results ofabovementioned step 4, reading the results of depthwise convolutioncalculations of p points, m channels from the register or the on-chipSRAM, and calculating the multiplication and accumulation results of ppoints, m channels, the accumulation here being the accumulation in thechannel direction, to obtain the pointwise convolution partial sums of ppoints, 1 output channel (the partial sums from the first to m-th inputchannels).

6. Storing the results of abovementioned step 5 in the register oron-chip SRAM.

7. Completing the calculation and storage of the pointwise convolutionpartial sums of p points, N output channels (the partial sums of thefirst to m-th input channels) by circulating N times of operations ofstep 5 and step 6, assuming that pointwise convolution have a total of nconvolution kernels, the results of the partial sums of the p outputpoints on the N output channels being stored in the register or on-chipSRAM.

As shown in FIG. 10, the size of the convolution kernel of the Pointwiseconvolution is 1*1, and it is also assumed that there are 512multiplier-adder units (the 512 multiplier-adder units here and the 512multiplier-adder units of the Depthwise calculation process may be thesame 512 multiplier-adder units, or may be different), and a convolutionof calculates 32 points (the abovementioned p=32), 16 channels (theabovementioned m=16, i.e. the channel variable c takes the value fromthe range of 0 to 15), 1 convolution kernel each time, and obtains thedata of a partial sums of 32 points on 1 output channel. Assuming thatthere are 256 convolution kernels, then after 256*(128/16)=2048 times ofcalculations, the data of the partial sums of 32 points on 256 outputchannels are obtained.

8. Repeating the operations of steps 1-7, continuously calculating toobtain the partial sums of p points on next m output channels (thepartial sums on the m+1-th to 2m-th input channels), and to accumulatethese partial sums with the previously stored partial sums and store theaccumulation results in the register or on-chip SRAM, in this way,obtaining the final accumulation results of the pointwise convolution ofthe p output points on the N output channels through [M/m] times ofcirculation.

9. Performing an optional activation operation on the results ofabovementioned step 8, the activation operation referring to remappingthe numerical with a nonlinear function, the activation functionincluding but not limited to ReLu function, Sigmoid function, arctangent(tan) function, etc.

10. Performing an optional quantization operation on the results ofabovementioned step 9, the quantization operation referring to obtainingthe low-precision multiplication and accumulation results (usually 8bit) by shifting or multiplying and dividing the high-precisionmultiplication and accumulation results (usually 32 bit).

11. Repeating the above operations of steps 1-10 by continuouslycalculate next p points until the complete output feature map isobtained.

Compared with the specific example of the first embodiment and thespecific example of the third embodiment, it may be seen that if thedepthwise convolution has M channels (or when the number of channels isconsidered to be 1, it may also be regarded as the number), andpointwise convolution has N convolution kernels, then the formersolution needs to buffer p*M depthwise convolution result data which aregenerally quantized (8bi), and a buffer space of p*M*8 bits is needed.The latter solution needs to buffer p*N partial sums data which aregenerally high-precision and unquantized (32 bits), and then p*N*32 bitsof storage space is required. It may be seen that, in the typical casewhere the depthwise convolution results are quantized and the partialsums results are not quantized, if M>4n, the latter solution will savemore storage space, otherwise the former solution will save more storagespace.

Therefore, in an embodiment of the present disclosure, the convolutioncalculation method in a neural network may further comprise: comparingthe number of channels (or number) M of convolution kernels and thenumber N of pointwise convolution kernels; in response to M>4N,performing the convolution calculation method according to the firstembodiment of the present disclosure to calculate the output featurevalues, otherwise, selecting the convolution calculation methodaccording to the third embodiment of the present disclosure to calculatethe output feature values.

Exemplary Electronic Device

Hereinafter, an electronic device according to an embodiment of thepresent disclosure will be described with reference to FIG. 15.

FIG. 15 shows a block diagram of an electronic device according to anembodiment of the present disclosure.

As shown in FIG. 15, an electronic device 10 comprises one or moreprocessors 11 and memories 12.

The processor 11 may be any form of processing unit having dataprocessing capability and/or instruction execution capability, and maycontrol other assembly in the electronic device 10 to perform thedesired functions.

The memory 12 may comprise one or more computer program products whichmay comprise various forms of computer readable and writable storagemedia, such as a volatile memory and/or a non-volatile memory. Thevolatile memory may comprise, for example, a random access memory (RAM)and/or a cache, etc. The non-volatile memory may comprise, for example,a read only memory (ROM), a hard disk, a flash memory, etc. One or morecomputer program instructions may be stored on the computer readablestorage medium, and the processor 11 may run the program instructions toimplement the convolution calculation method and/or other desiredfunctions in the neural network of various embodiments of the presentdisclosure as described above.

In one example, the electronic device 10 may also comprise an inputdevice 13 and an output device 14, and these assembly are interconnectedby a bus system and/or other form of connection mechanism (not shown).

For example, the input device 13 may comprise, for example, a keyboard,a mouse, and a communication network and remote input devices to whichit is connected, and the like.

For example, the output device 14 may comprise, for example, a display,a printer, and a communication network and remote output devices towhich it is connected, and the like.

Of course, for simplicity, only some of the assemblies related to thepresent disclosure in the electronic device 10 are shown in FIG. 15, andassemblies such as a bus, an input/output interface, and the like areomitted. It should be noted that the assemblies and structures of theelectronic device 10 shown in FIG. 15 are merely exemplary and notlimiting, and the electronic device 10 may have other assemblies andstructures as needed.

Exemplary Computer Program Product and Computer Readable and WritableStorage Medium

In addition to the methods and apparatus described above, embodiments ofthe present disclosure may also be a computer program product whichcomprises computer program instructions, and said computer programinstructions, when executed by a processor, make the processor toperform steps in a composite operation method for a neural networkaccording to various embodiments of the present disclosure as describedin the abovementioned “exemplary method” portion of the presentdisclosure.

The computer program product may write program code for performingoperations of embodiments of the present disclosure in any combinationof one or more programming languages which comprise object-orientedprogramming languages, such as Java, C++, etc., and conventionalprocedural programming languages, such as “C” language or similarprogramming languages. The program code may be executed entirely on auser computing device, be partially executed on a user device, beexecuted as a stand-alone software package, be partially executed on auser computing device and be partially executed on a remote computingdevice, or be entirely executed on a remote computing device or server.

Furthermore, embodiments of the present disclosure may also be acomputer readable and writable storage medium having computer programinstructions stored thereon, and said computer program instructions,when executed by a processor, make the processor to perform steps in acomposite operation method for a neural network according to variousembodiments of the present disclosure as described in the abovementioned“exemplary method” portion of the present disclosure.

The computer readable and writable storage medium may use anycombination of one or more readable and writable media. The readable andwritable medium may be a readable and writable signal medium or areadable and writable storage medium. The readable and writable storagemedium may comprise, but are not limited to, an electric, a magnetic, anoptical, an electromagnetic, an infrared, or a semiconductor system,apparatus, or device, or any combination of the above. More specificexamples (a non-exhaustive list) of readable and writable storage mediuminclude an electrical connection with one or more wires, a portabledisk, a hard disk, a random access memory (RAM), a read only memory(ROM), an erasable programmable read only memory (EPROM or a flashmemory), an optical fiber, a portable compact disk read only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the above.

The basic principles of the present disclosure are described above inconjunction with the specific embodiments. However, it is necessary topoint out that the advantages, superiorities, and effects and so onmentioned in the present disclosure are merely examples but not intendedto limit the present invention. These advantages, superiorities, effectsand so on will not be considered as essential to the embodiments of thepresent disclosure. In addition, the specific details of the foregoingdisclosure are only for the purpose of illustration and ease ofunderstanding but not for the purpose of limitation, and the abovedetails do not limit the application to be implemented in the specificdetails mentioned above.

The block diagrams of device, apparatus, equipment, system referred toin the present disclosure are merely illustrative examples and are notintended to require or imply that the connections, arrangements, andconfigurations must be made in the manner shown in the block diagram. Itwill be appreciated by those skilled in the art that the device,apparatus, equipment, system, may be connected, arranged, or configuredin any manner. Terms such as “including”, comprising”, “having” and thelike are open words, which means “including but not limited to” and maybe used interchangeably. The terms “or” and “and” as used herein referto the term “and/or” and may be used interchangeably, unless the contextclearly dictates otherwise. The term “such as” as used herein refers tothe phrase “such as but not limited to” and is used interchangeably.

It should also be noted that in the apparatus, equipment, and the methodof the present disclosure, each component or each step may be decomposedand/or recombined. These decompositions and/or recombination should beregarded as an equivalent of the present disclosure.

The above description of the disclosed aspects is provided to enable anyof those skilled in the art to make or use the application. Variousmodifications to these aspects are very obvious for those skilled in theart, and the generic principles defined herein may be applied to otheraspects without departing from the scope of the application. Therefore,the present disclosure is not intended to be limited to the aspectsshown herein, but rather to present the broadest scope consistent withthe principles and novel features disclosed herein.

The above description has been provided for the purposes of illustrationand description. In addition, the description is not intended to limitthe embodiments of the present disclosure to the forms disclosed herein.Although various exemplary aspects and embodiments have been discussedabove, those skilled in the art will recognize some variant,modification, changes, addition and sub-combination.

What is claimed is:
 1. A method for convolution calculation in a neuralnetwork, comprising: reading an input feature map, depthwise convolutionkernels and pointwise convolution kernels from a dynamitic random accessmemory (DRAM); performing depthwise convolution calculations andpointwise convolution calculations according to the input feature map,the depthwise convolution kernels and the pointwise convolution kernelsto obtain output feature values of a first predetermined number p ofpoints on all pointwise convolution output channels; storing the outputfeature values of a first predetermined number p of points on allpointwise convolution output channels into an on-chip memory, whereinthe first predetermined number p is determined according to at least oneof available space in the on-chip memory, a number of the depthwiseconvolution calculation units, height and width, and channel numberdimensions of the input feature map; and repeating the above operationto obtain output feature values of all points on all pointwiseconvolution output channels.
 2. The method of claim 1 wherein theperforming step comprises: performing the depthwise convolutioncalculations according to the input feature map and the depthwiseconvolution kernels to obtain intermediate feature values of the firstpredetermined number p of points on all depthwise convolution outputchannels; and performing the pointwise convolution calculationsaccording to the intermediate feature values of the first predeterminednumber p of points on all depthwise convolution output channels and thepointwise convolution kernels, to obtain the output feature values ofthe first predetermined number p of points on all pointwise convolutionoutput channels.
 3. The method of claim 2 wherein performing thedepthwise convolution calculations according to the input feature mapand the depthwise convolution kernels to obtain the intermediate featurevalues of the first predetermined number p of points on all depthwiseconvolution output channels comprises: performing the depthwiseconvolution calculations according to the input feature map and thedepthwise convolution kernels, to obtain intermediate feature values ofthe first predetermined number p of points on a second predeterminednumber m of depthwise convolution output channels, wherein the secondpredetermined number m is determined according to a number of depthwiseconvolution calculation units and the first predetermined number p; andrepeating the above operations by performing the depthwise convolutioncalculations according to the input feature map and the depthwiseconvolution kernels to obtain intermediate feature values of the firstpredetermined number p of points on a next second predetermined number mof depthwise convolution output channels and correspondingly performingsubsequent operations, until the intermediate feature values of thefirst predetermined number p of points on all depthwise convolutionaloutput channels are obtained.
 4. The method of claim 3 whereinperforming the depthwise convolution calculations according to the inputfeature map and the depthwise convolution kernels to obtain intermediatefeature values of the first predetermined number p of points on a secondpredetermined number m of depthwise convolution output channelscomprises: reading input feature values of the first predeterminednumber p of groups of points on the second predetermined number m ofinput channels from the input feature map; reading weight values in acorresponding second predetermined number m of depthwise convolutionkernels corresponding to the input feature values of the firstpredetermined number p of groups of points on the second predeterminednumber m of input channels; and respectively performing the depthwiseconvolution calculations on the input feature values of the firstpredetermined number p of groups of points on the second predeterminednumber m of input channels with the weight values in the correspondingsecond predetermined number m of depthwise convolution kernels, toobtain the intermediate feature values of the first predetermined numberp of points respectively corresponding to the first predetermined numberp of groups of points on the second predetermined number m of depthwiseconvolution output channels.
 5. The method of claim 4, furthercomprising: concurrent with the step of performing the pointwiseconvolution calculations, reading the input feature values of the firstpredetermined number p of groups of points on the second predeterminednumber m of input channels from the input feature map and thecorresponding weight values.
 6. The method of claim 4 wherein readinginput feature values of the first predetermined number p of groups ofpoints on the second predetermined number m of input channels from theinput feature map comprises: reading the input feature values of thefirst predetermined number p of groups of points on the secondpredetermined number m of input channels from the input feature map,each group of points having a width and a height equal to a width and aheight of the weight values in the depthwise convolution kernel, and twoadjacent groups of points having a stride equal to a stride of thedepthwise convolution calculation.
 7. The method of claim 4 whereinrespectively performing the depthwise convolution calculations on theinput feature values of the first predetermined number p of groups ofpoints on the second predetermined number m of input channels with theweight values in the corresponding second predetermined number m ofdepthwise convolution kernels to obtain intermediate feature values ofthe first predetermined number p of points respectively corresponding tothe first predetermined number p of groups of points on the secondpredetermined number m of depthwise convolution output channelscomprises: performing following operations on each group of points inthe first predetermined number p of groups of points: respectivelyperforming multiplication calculations on input feature values of onepoint in the group of points on the second predetermined number m ofinput channels and a corresponding weight value in the correspondingsecond predetermined number m of depthwise convolution kernels, toobtain current multiplication calculation results of the group ofpoints; respectively performing accumulation calculations on the currentmultiplication calculation results of the group of points and previousmultiplication calculation results of the group of points, the previousmultiplication calculation results being obtained by respectivelyperforming multiplication calculations on the input feature values of aprevious point in the group of points on the second predetermined numberm of input channels and a corresponding previous weight value in thecorresponding second predetermined number m of depthwise convolutionkernels; and repeating the above operations by respectively performingmultiplication calculations on input feature values of a next point inthe group of points on the second predetermined number m of inputchannels and a corresponding next weight value in the correspondingsecond predetermined number m of depthwise convolution kernels andcorrespondingly performing subsequent operations, until themultiplication and accumulation operations are completed on the inputfeature values of all points in the group of points on the secondpredetermined number m of input channels, the final accumulationcalculation results of the group of points being the intermediatefeature values of one point corresponding to the group of points on thesecond predetermined number m of depthwise convolution output channels.8. The method of claim 3, further comprising: storing each intermediatefeature value in the on-chip memory after obtaining the intermediatefeature values of the first predetermined number p of points on thesecond predetermined number m of depthwise convolution output channels.9. The method of claim 8, further comprising: performing at least one ofan activation operation and a quantization operation on eachintermediate feature value before storing it in the on-chip memory. 10.The method of claim 2 wherein performing the pointwise convolutioncalculations according to intermediate feature values of the firstpredetermined number p of points on all depthwise convolution outputchannels and the pointwise convolution kernels, to obtain output featurevalues of the first predetermined number p of points on all pointwiseconvolution output channels comprises: performing the pointwiseconvolution calculations according to the intermediate feature values ofthe first predetermined number p of points on all depthwise convolutionoutput channels and weight values on all pointwise convolution channelsin a fourth predetermined number n of pointwise convolution kernels,respectively, to obtain output feature values of the first predeterminednumber p of points on a fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels; and repeating the aboveoperations by performing the pointwise convolution calculationsaccording to the intermediate feature values of the first predeterminednumber p of points on all depthwise convolution output channels andweight values of a next fourth predetermined number n of pointwiseconvolution kernels on all pointwise convolution channels, respectively,until obtaining output feature values of the first predetermined numberp of points on all pointwise convolution output channels.
 11. The methodof claim 10 wherein performing the pointwise convolution calculationsaccording to the intermediate feature values of the first predeterminednumber p of points on all depthwise convolution output channels andweight values of a fourth predetermined number n of pointwiseconvolution kernels on all pointwise convolution channels, respectively,to obtain output feature values of the first predetermined number p ofpoints on a fourth predetermined number n of pointwise convolutionoutput channels corresponding to the fourth predetermined number n ofpointwise convolution kernels comprises: reading intermediate featurevalues of the first predetermined number p of points on a thirdpredetermined number m′ of depthwise convolution output channels fromthe intermediate feature values of the first predetermined number p ofpoints on all depthwise convolution output channels; reading weightvalues of a fourth predetermined number n of pointwise convolutionkernels on a corresponding third predetermined number m′ of pointwiseconvolution channels corresponding to the intermediate feature values ofthe first predetermined number p of points on the third predeterminednumber m′ of depthwise convolution output channels; respectivelyperforming the pointwise convolution calculations on the intermediatefeature values of the first predetermined number p of points on thethird predetermined number m′ of depthwise convolution output channelswith the weight values on the corresponding third predetermined numberm′ of pointwise convolution channels in the fourth predetermined numbern of pointwise convolution kernels, to obtain current pointwiseconvolution partial sums of the first predetermined number p of pointson the fourth predetermined number n of pointwise convolution outputchannels corresponding to the fourth predetermined number n of pointwiseconvolution kernels; respectively performing accumulation calculationson the current pointwise convolution partial sums of the firstpredetermined number p of points on the fourth predetermined number n ofpointwise convolution output channels corresponding to the fourthpredetermined number n of pointwise convolution kernels and previousaccumulation calculation results of the first predetermined number p ofpoints, to generate current accumulation calculation results of thefirst predetermined number p of point, wherein the third predeterminednumber m′ and the fourth preset number n are determined according to anumber of pointwise convolution calculation units and the firstpredetermined number p; and repeating the above operations by readingintermediate feature values of the first predetermined number p ofpoints on a next third predetermined number m′ of depthwise convolutionoutput channels, reading weight values of the fourth predeterminednumber n of pointwise convolution kernels on a corresponding next thirdpredetermined number m′ of pointwise convolution channels, andcorrespondingly performing subsequent operations until the pointwiseconvolution calculations and accumulation calculations are completed onthe intermediate feature values of the first predetermined number p ofpoints on all depthwise convolution output channels, the finalaccumulation calculation results of the first predetermined number p ofpoints being the output feature values of the first predetermined numberp of points on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels.
 12. The method of claim 10wherein performing the pointwise convolution calculations on theintermediate feature values of the first predetermined number p ofpoints on the third predetermined number m′ of depthwise convolutionoutput channels with the weight values on the corresponding thirdpredetermined number m′ of pointwise convolution channels in the fourthpredetermined number n of pointwise convolution kernels, to obtaincurrent pointwise convolution partial sums of the first predeterminednumber p of points on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels comprises: performingfollowing operations on each point of the first predetermined number pof points: respectively performing multiplication calculations on theintermediate feature values of the point of the first predeterminednumber p of points on the third predetermined number m′ of depthwiseconvolution output channels with the weight values on the correspondingthird predetermined number m′ of pointwise convolution channels in thefourth predetermined number n pointwise convolution kernels, to obtain afourth predetermined number n of groups of results, each group of whichcomprise a third predetermined number m′ of multiplication calculationresults; and respectively adding the third predetermined number m′ ofmultiplication calculation results of each group of results from thefourth predetermined number n of groups of results, to obtain currentpointwise convolution partial sums of the point on the fourthpredetermined number n of pointwise convolution output channelscorresponding to the fourth predetermined number n of pointwiseconvolution kernels.
 13. The method of claim 11, further comprising:storing the current accumulation calculation results of the firstpredetermined number p of points in the on-chip memory after generatingthe current accumulation calculation results, to cover the previousaccumulation calculation results of the first predetermined number p ofpoints.
 14. The method of claim 13, further comprising: performing atleast one of an activation operation and a quantization operation oneach output feature values of the first predetermined number p of pointson the fourth predetermined number n of pointwise convolution outputchannels corresponding to the fourth predetermined number n of pointwiseconvolution kernels before storing the final accumulation calculationresults of the first predetermined number p of points in the on-chipmemory as the output feature values.
 15. The method of claim 2, whereinperforming the pointwise convolution calculations according to theintermediate feature values of the first predetermined number p ofpoints on all depthwise convolution output channels and the pointwiseconvolution kernels, to obtain the output feature values of a firstpredetermined number p of points on all pointwise convolution outputchannels comprises: performing the pointwise convolution calculationsaccording to intermediate feature values of the first predeterminednumber p of points on a third predetermined number m′ of depthwiseconvolution output channels and weight values of all pointwiseconvolution kernels on a corresponding third predetermined number m′ ofpointwise convolution channels, respectively, to obtain a currentpointwise convolution partial sums of the first predetermined number pof points on all pointwise convolution output channels; respectivelyperforming accumulation calculations on the current pointwiseconvolution partial sums of the first predetermined number p of pointson all pointwise convolution output channels and previous accumulationcalculation results of the first predetermined number p of points, togenerate current accumulation calculation results of the firstpredetermined number p of points; and repeating the above operations byperforming the pointwise convolution calculations according tointermediate feature values of the first predetermined number p ofpoints on a next third predetermined number m′ of depthwise convolutionoutput channels and weight values of all pointwise convolution kernelson a corresponding next third predetermined number m′ of pointwiseconvolution channels, respectively, and correspondingly performingsubsequent operations, until all of the intermediate feature values ofthe first predetermined number p of points on all depthwise convolutionoutput channels completing pointwise convolution calculations andaccumulation calculations, final accumulation calculation results of thefirst predetermined number p of points being the output feature valuesof the first predetermined number p of points on all pointwiseconvolution output channels.
 16. The method of claim 15, whereinperforming the pointwise convolution calculations according tointermediate feature values of the first predetermined number p ofpoints on a third predetermined number m′ of depthwise convolutionoutput channels and weight values in all pointwise convolution kernelson a corresponding third predetermined number m′ of pointwiseconvolution channels, respectively, to obtain a current pointwiseconvolution partial sums of the first predetermined number p of pointson all pointwise convolution output channels comprises: performing thepointwise convolution calculations according to the intermediate featurevalues of the first predetermined number p of points on the thirdpredetermined number m′ of depthwise convolution output channels andweight values of a fourth predetermined number n of pointwiseconvolution kernels on a corresponding third predetermined number m′ ofpointwise convolution channels, to obtain a current pointwiseconvolution partial sums of the first predetermined number p of pointson a fourth predetermined number n of pointwise convolution outputchannels corresponding to the fourth predetermined number n of pointwiseconvolution kernels, and repeating the above operations by performingthe pointwise convolution calculations on the intermediate featurevalues of the first predetermined number p of points on the thirdpredetermined number m′ of depthwise convolution output channels andweight values of a next fourth predetermined number n of pointwiseconvolution kernels on the corresponding third predetermined number m′of pointwise convolution channels, to obtain a current pointwiseconvolution partial sums of the first predetermined number p of pointson a next fourth predetermined number n of pointwise convolution outputchannels corresponding to a next fourth predetermined number n ofpointwise convolution kernels, until obtaining a current pointwiseconvolution partial sums of the first predetermined number p of pointson all pointwise convolutional output channels.
 17. The method of claim16, wherein performing the pointwise convolution calculations accordingto the intermediate feature values of the first predetermined number pof points on the third predetermined number m′ of depthwise convolutionoutput channels and weight values of all pointwise convolution kernelson the corresponding third predetermined number m′ of pointwiseconvolution channels, respectively, to obtain a current pointwiseconvolution partial sums of the first predetermined number p of pointson all pointwise convolution output channels comprises: reading theintermediate feature values of a first predetermined number p of pointson the third predetermined number m′ of depthwise convolution outputchannels from the intermediate feature values of the first predeterminednumber p of points on all depthwise convolution output channels; readingweight values of a fourth predetermined number n of pointwiseconvolution kernels on the corresponding third predetermined number m′of pointwise convolution channels correspondingly to intermediatefeature values of the first predetermined number p of points on thethird predetermined number m′ of depthwise convolution output channels;and performing the pointwise convolution calculations on theintermediate feature values of the first predetermined number p ofpoints on the third predetermined number m′ of depthwise convolutionoutput channels with the weight values in the fourth predeterminednumber n of pointwise convolution kernels on the corresponding thirdpredetermined number m′ of pointwise convolution channels, to obtain acurrent pointwise convolution partial sums of the first predeterminednumber p of points on the fourth predetermined number n of pointwiseconvolution output channels corresponding to the fourth predeterminednumber n of pointwise convolution kernels.
 18. The method of claim 15,further comprising: storing the current accumulation calculation resultsof the first predetermined number p of points in the on-chip memory tocover the previous accumulation calculation results of the firstpredetermined number p of points after generating the currentaccumulation calculation results.
 19. The method of claim 18, furthercomprising: performing at least one of an activation operation and aquantization operation on each output feature value of the firstpredetermined number p of points on all pointwise convolution outputchannels before storing the final accumulation calculation results ofthe first predetermined number p of points in the on-chip memory as theoutput feature values.
 20. An electronic device, comprising: aprocessor; and a memory having computer program instructions storedtherein, when executed by the processor, making the processor to performa method of convolution calculation method in a neural networkcomprising: reading an input feature map, depthwise convolution kernelsand pointwise convolution kernels from a dynamitic random access memory(DRAM); performing depthwise convolution calculations and pointwiseconvolution calculations according to the input feature map, thedepthwise convolution kernels and the pointwise convolution kernels toobtain output feature values of a first predetermined number p of pointson all pointwise convolution output channels; storing the output featurevalues of a first predetermined number p of points on all pointwiseconvolution output channels into an on-chip memory, wherein the firstpredetermined number p is determined according to at least one ofavailable space in the on-chip memory, a number of the depthwiseconvolution calculation units, and width, height and channel dimensionsof the input feature map; and repeating the above operation to obtainoutput feature values of all points on all pointwise convolution outputchannels.